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Sir: 

We, Ruth Rosenholtz, Allison G. Woodruff and Andrew Faulring, hereby declare and 
state that: 

1 . This Declaration is submitted as evidence that the invention of this application 
was invented by us prior to June 15, 2001, which is the effective filing date of U.S. Patent 
Application Publication No. 2003/0006995 to Smith et aL, entitled "Orthogonal Magnifier 
Within a Computer System Display," which was applied in the February 16, 2005 Office 
Action. 

2. We are the named inventors in the above- identified application. 

3. We are the inventors of the invention described in an invention proposal 
entitled "Enhanced Thumbnails of Documents," which appears as Exhibit A attached to this 
Declaration. The invention proposal is signed by us and dated October 3, 2000. 

4. The copies of these pages which constitute Exhibit A are true copies of the 
invention proposal 
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5. The invention described by Exhibit A was conceived and actually reduced to 
practice by us in the United States at least as early as October 3, 2000. 

6. We were in possession of the invention recited in claims 1-3, 6-22, 25-28 and 
30-40 as evidenced by the entire disclosure of Exhibit A. 

7. The features recited in claims 1, 16, 20, 21, 28, 34 and 38 are generally 
described at, for example, page 2, lines 15-17, and page 3, lineB 16-21, and shown in Fig. 1. 
The features recited in claims 2, 3, 10, 11, 17, 22 and 26 are generally described at, for 
example, page 2, lines 33-37. Hie features recited in claims 6, 9, 12, 26 and 27are generally 
described at, for example, page 2, lines 25-30. The features recited in claims 7, 8, 13- IS and 
30-33 are generally described at, for example, page 3, lines 6-15. The features recited in 
claim 18 are generally described at, for example, page 3, lines 3-5. The features recited in 
claims 19 and 37 are generally described at, for example, page 2, line 38-page 3, line 2. The 
features recited in claims 35 and 36 arc generally described at, for example, page 2, line 33- 
page 3, line 15. The features recited in claims 40 and 41 are generally described at, for 
example, page 2, lines 6-10. 
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8. We hereby declare and state that all statements made herein of our own 

knowledge are true, and that all statements made on information and belief are believed to be 

true; and further that these statements were made with the knowledge that willful false 

statements and the like so made are punishable by fine and/or imprisonment under Section 

1001 of Title 18 of the United States Code, and that such willful false statements may 

jeopardize the validity of the application or any patent issuing therefrom. 



Date: 5/22/2005 



Ruth Rosenholtz 

Date: 



Allison Gyle Woodruff 
Date: 



Andrew Fauling 
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Many search engine vendors could be interested in using this technology, among otners. uur siugy sx.uw* ,.«., k 
are significantly more efficient searching web pages when using our enhanced thumbnails versus traditional text 
summaries or enhanced thumbnails. — 



Descriptive title of invention 

Enhanced thumbnails of documents 



Describe the problem How was this problem tackled before your invention? 

The challenge is to create thumbnails that are information-rich yet retain the ability to be scanned quickly. Previous 
thumbnails fall into two categories: (1) simple scale-reduced images of documents [e.g., 1, 2, 3] (occasionally augmented 
with metadata about the document [4]) and (2) images composed of selected elements or visual representations of 
metadata properties from the document [5]. The former contain little information, e.g., often none of the text in the 
thumbnail can be read. The latter images bear little visual resemblance to the original document, so properties like genre 
that can be scanned quickly are lost. It remains to be seen how quickly people can scan abstract representations of 
metadata. 



Summary of the invention Describe briefly what the invention is and how it works in 5 -8 lines. _~ .... , K * 

We introduce a technique for creating novel, enhanced thumbnails of documents. Theselhumbnaila combine Ine 
advantages of image thumbnails and of other types of summaries (e.g. text), i.e.. they can be acenned quicKry but oonwn 
more information than traditional thumbnails. Our thumbnails include a reduced imago ol (he documeni elonQ w»ih 
various forms of emphasis of information in the document: (1) modifying the document format (e.g., Hi ML) pfioi lo «c«i* 
reduction or (2) overlaying graphical elements (e.g.. text callouts) on the scale-reduced Ihumbnail. 
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Describe your invention Describe how to make and use the invention and it's novel embodiments. Cover the process, method, materials with 
sketches, flow charts, usage etc. What are the advantages of your invention for Xerox? 

In this IP we describe our enhanced thumbnails, focusing on a particular instantiation of those thumbnails that we have 
tested and found to be more effective for a web search task than either text summaries or simple reduced-image 
thumbnails. We also describe, in brief, the system for generating these thumbnails, and in the process try to give a 
feeling for the breadth of what we mean by "enhanced thumbnails." For more details, we include our recent CHI 
submission, "Using Thumbnails to Search the Web. B 

Our enhanced thumbnails could be useful in virtually any system in which thumbnails might be useful. Some suggested 
applications are browsing or searching the web, management of email or an on-line document corpus, and so on. 
Though we have so far implemented a system that generates a thumbnail from an HTML document, one could generate 
enhanced thumbnails from virtually any text/mixed-media file format, and possibly even from an image format (e.g. a 
scanned document) plus OCR. 

Generating enhanced thumbnails is a multi-step process involving multiple renderings. The steps include the following, or 
some subset thereof: 

1 . Specify important elements of the document. 

2. Modify the original document format. 

3. Render a reduced image of the modified document. 

4. Extract properties of the reduced image. 

5. Re-render the thumbnails with one or more overlays, possibly making use of the extracted properties. 

Below we discuss these steps in somewhat more detail, giving examples of the resulting thumbnails. We believe 
thumbnails generated using such a , 
series of steps to be novel, as they both allow a great deal of preservation of the original document format - which 
provides information about such things as document genre - and the presentation of additional information in the form of 
overlays on the basic thumbnails. Previous thumbnails have either been simple reduced images of a page of the 
document, possibly including abstract representations of document metadata, or abstract collections of elements or 
metadata from the document, represented in a way that loses all of the original appearance of the document. 

Specify important elements of the document: 

There are a number of ways that the important elements might be specified. The user might specify them, in some 
applications. In our implementation, which is a web search application, we took search terms and hi and h2 headers to 
indicate important text. One might also imagine using text words with high TF*IDF scores, terms resulting from 
information scent computations [6], the document title or URL, and so on. Items other than text might be labelled as 
important: images labelled as "representative" [5], tables, etc. 

In general, items could be specified as important elements by the output of any model of importance in a document: 
models of user intent, models of human perception, eye-tracking data, etc. 

Modify the format of the original document: 

Here we change the formatting of the original document, along any dimension in which one typically has control of the 
formatting In our implementation we increase the sjz|of hi headers so as to make them "readable" in the reduced 
thumbnail image (we specify the size required for Feasibility by hand, but this might also be done automatically in various 
ways). We also highlight important words specified by the previous step. 

Render a reduced-size version of the modified document: 

In our implementation, we render a simple reduced image of either the first page of the document, a page of the 
document starting at an arbitrary point in the document, or the full document. Among other variants one might imagine 
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rendering with an arbitrary affine transformation or distortion of the document [8]. This step could make use of prior art on 
better filtering and subsampling techniques for generating reduced document images [e.g. 9]. 

Extract properties of the reduced image: 

In our implementation, this includes extracting statistics of the colors in the reduced image, and the location and size of 
selected elements within the reduced image. 

Re-render the thumbnails with one or more overlays: 

In our implementation, we first overlay a translucent "fill" color over the entire thumbnail. This desaturates the thumbnail 
colors. This decreases the distraction from attention-grabbing elements in the original document, allowing our overlays to 
better'draw attention, yet still allowing the basic structure of the original document to show through. We also overlay 
enlarged words from step 1 , centered on the location of the same text in the underlying thumbnail. In our implementation, 
this text is highlighted by one of several translucent colors, each word consistently highlighted with the same color. We 
selected the highlight colors in a principled way, based upon a model of human vision, but they might also be selected by 
hand or in another way. Our implementation allows these highlighted words to (optionally) extend over the edge of the 
thumbnail. Furthermore, in our implementation these overlaid words maintain their original format except where specified, 
so that, e.g., HTML links still look like links (blue, underlined). 

Figure 1 shows a number of examples of thumbnails we have generated that satisfy this constructive definition of 
enhanced thumbnails. As described in the paper, we have tested one variant of enhanced thumbnails in a web search 
task, comparing with performance using standard text summaries or plain reduced-image thumbnails. We found that 
users were significantly faster at finding the answer to their query using the enhanced thumbnails. Because of this result 
and our own experiences using the thumbnails, we believe that at the very least this invention may provide a licensing 
opportunity for Xerox to some web search company. 

One area we have not yet explored is the use of animation in the thumbnails. For example, the thumbnails might animate 
to emphasize different pieces of information at different times, e.g., on mouse-over. 

In addition to the novelty of the enhanced thumbnails, we believe there may be novelty in a couple of navigation system 
improvements suggested by the enhanced thumbnails. We have implemented neither of these system enhancements. 

We call the first "click-through." Normally, when a user clicks with a mouse on a thumbnail, we'd like them to go to the 
document represented by the thumbnail. With click-thro ugh, the user would click in a particular way on a particular part of 
the thumbnail, and would either be brought toTfie corresponding part of the document, or would be brought to a document 
linked to by the corresponding link in the original document. Which of the two actions occurred might depend both on 
where the user clicked (is there a link in that location?) and how they clicked (middle button vs. right, keypress or no, etc.). 
This system enhancement is enabled by the enhanced thumbnails, since they may enhance text that will guide the user to 
a particular location in the document, and also may enhance links so that they are visible, obviously links, and large 
enough to click on easily. 

The second system enhancement suggested by our thumbnails is also to enhance the document, at least temporarily. In 
this addition to the system, a user might click on an enhanced thumbnail and be taken to a document enhanced in the 
same way. This consistency between thumbnail and destination document should help the user find the document 
elements that drew their interest in the thumbnail. The enhanced document might, after a short time, morph to the 
original document. 
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Figure 1 . Top row (from left to right): Plain thumbnail. Thumbnail enhanced with HTML modification. Thumbnail enhanced with HTML 
and image modification. E-commerce genre example. Bottom row (from left to right): News genre example. Homepage genre 
example. Plain thumbnail of text page. Enhanced thumbnail of text page. 
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People List names of others known to have worked on this or a similar invention 
Examples o f plain th umbnalls: 

[1] Ayers, E. and Stasko, J. Using Graphic History in Browsing the World Wide Web. In Proc. 4th Intl. WWWConU 
Boston, December 1995. 

[2] Hightower, R., Ring, L, Helfman, J., Bederson, B., and Hoilan, J. Graphical Multiscale Web Histories: A Study of 
PadPrints. In Proc. Hypertext'98, 1998, 58-65. 

[3] Kopetzky, T. and Muhihauser, M. Visual Preview for Link Traversal on the WWW. In Proc. 8th Intl. WWW Conference, 
Toronto, Canada, May 1999, 447-454. 

Plain thumbnails plus metadata: 

[A] Cockburn, A., et al. WebView: A Graphical Aid for Revisiting Web Pages. In Proc. OZCHl'99 Australian Conf. on HCI, 
Wagga Wagga, Australia, Nov. 1 999. 

Thumbnail "caricatures' - selected elements plus metadata, not attempting to mimic the appearance of the document: 

[5] Wynblatt, M. and Benson, D. Web Page Caricatures: Multimedia Summaries for WWW Documents. In Proc. IEEE Intl. 
Conf. on Multimedia Computing and Systems, Austin, TX, June, 1 998, 1 94-1 99. 

Infomnation scent: 

[6].OIston, C. and Chi, E. ScentTraiis: Integrating Browsing and Searching on the World Wide Web. Submitted to 
SIGCHI 2001. 

Fish-eye views: 

[7] Card, S. K., Mackinlay, J. D., and Shneiderman, B. Information Visualization: Using Vision to Think. Morgan- 
Kaufmann, San Francisco, 1998. 

[8] Patent #5740285, Bloomberg, Dan S. and Davies, Daniel. Image reduction/enlargement technique. 



Related concepts Check the Xerox Patent data base at httD://comiP.wrC.xerOX.COm/comip/icbuhome.nsf 

What have you found in a data base search of the topic? Give patent or IP number of the most relevant items. 
Patent #5526443, Nakayama, Takehiro. Method and apparatus for highlighting and categorizing documents using coded 
word tokens. 

Patent #5384863, Huttenlocher, Daniel P. et al. Methods and apparatus for automatic modification of semantically 
significant portions of a document without document image decoding. 

Patent #5060135, Levine et. al. Apparatus for manipulating documents in a data processing system utilizing reduced 
images of sheets of information which are movable 



Prototype Has a model, a prototype, or experiment of the invention been built, made, run or tested ? )8 Yes □ No 

Yes. A prototype has been written in Java, using a component web browser (ICE Browser 5). 



□ Yes IS No 



Xerox product Is the invention used by Xerox or is there a definite plan for use in a future product(s) ? 
If so, please identify the program(s) or product(s), and introduction dates: 
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Disclosures Has this conce pt been disclosed to vendors, consultants, outside parties, partners, etc? Indicate the date(s) of any previous or planned 
future disclosure external to Xerox, and identify the type of disclosure (by agreement, demonstration, paper or presentation given, market 

was:receh% 



Outside funding H YES (Indicate Source of outside funding) □ N0 

<yesff artial funding for portions of the project were received by Office of Naval Research Contract No. N00014-96-C-" 
-0097: " ' 
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(Touch the hi-lighted areas to receive a definition of the category.) 



Architecture and Document Services 

1 .1 Advanced Print Serv ices 

1.2 Document Access & Management 

1.3 Document Capture & Analysis 

1.4 Document Systems Architecture 

1.5 Electronic Document Commerce 

1.6 Networked Document Systems 

1.7 Productivity Initiatives 

1.8 Process, Workflow, Information Management 

1.9 Smart Design & Service 

1.10 Work Process Analysis 



Digital Imaging 

2.1 Capture 

2.2 Presentation 

2.3 Manipulation 

2.4 Representation 

2.5 Systems 



Marking & Devices 

3.1 Latent Image Formation (Re-lmageable Process) 
Development (Re-lmageable Process) 
Image Transfer & Fixing (Re-lmageabie Process) 
Erase And Cleaning (Re-lmageable Process) 
Fixed Image Marking (Incl. Direct To Plate) 
Imager (ROS, Optics, Modulator, Illumination) 
Thermal Ink Jet 
Acoustic Ink Jet 
Continuous Ink Jet 
On Demand Powder 
Other Direct Marking 

3.12 Controls & Diagnostics (For Marking Systems) 

3.13 Media Handling (Feeding, Transport, Finishing) 

3.14 Marking System Integration & Architecture 

3.15 Marking Hybrid Processes 

3.16 Display Devices 

3.17 MEMS Devices 

3.18 Data Recording Devices 

3.19 Digital Image Scanning 



3.2 

3.3 

3.4 

3.5 

3.6 

3.7 

3.8 

3.9 

3.10 

3.11 



Materials & Materials Manufacturing 

4.1 Toner, Developer and Components (For Re-lmageable 
Process) 

4.2 Photoreceptors and Components 

4.3 Dielectric Receivers 

4.4 Inks For Direct Marking 

4.5 Powders For Direct Marking 

4.6 Substrate Media (Paper, Transparencies, etc.) 

4.7 Electronic Materials (Light Emitting Or Detecting, 
Semiconductors For Printhead Or Other Use) 

4.8 Display Materials 

4.9 Materials for Fusing 

4.10 Drum And Belt Substrates 

4.11 Materials for Binding and Finishing 

4.12 Materials of Controlled Conductivity 

4.13 Transfix Belt 

4.14 Intermediate Transfer Belts 

4.15 Magnetic Materials 

4.16 Recording Media 

4.17 Packaging Materials 



Manufacturing Technology & Product Elements 

5.1 Component Development 

5.2 Manufacturing Processes 

5.3 Production Systems 

5.4 Industrial Design / Human Factors 

5.5 Device Electronics 

5.6 Product Packaging 



Speculative Research 

6.1 Document Futures 

6.2 Applications outside Defined Xerox Direction 
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Submitter(s): 


Title of Invention 


Manager's Name 


Date 


1. Problem addressed or function provided by the invention: 
Example 1A: Finisher cost reduction 


Example 1B: Uses low cost LCD to write annotation messages 


2. Central thrust of the invention: 

Example 2A: Design incorporates fewer parts 


Example 2B: Uses tow cost LCD to write annotation messages 


3. Could invention have impact beyond current description? 
Example 3A: Could also function for printer finisher 


Example 3B: Could also function to erase/edit copy 


4. Potential for Xerox application. Specify product or technology program if possible: 

Example 4A: Mainline approach in Program Q Example 4B: Adds significant feature to future products 


5. Value to competitors; potential for license or trade: 

Example 5A: Enables much lower cost finishing than any known system 
and opens possibilities of moving finishing down-market 


Example SB: Could be licensed in a business area un-related to Xerox 


6. Please indicate any related patents, publications, or activities you know of: 


7. I would recommend the following form(s) of protection: □ Patent 


□ Defense publication □ Keep trade secret □ None 


Comments: 
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